
Cyberdog Usability Testing 0 Round 4 

This report documents usability testing performed on the d3 build of Cyberdog. This testing took place 
on Thursday, September 16, and Friday, September 17, in the Piaget's Kids usability labs in R&D2. For 
more information on testing contact Pat Holleran {holleran@apple.com, 974-1427). 



Testing Goals 

Goals of the present testing protocols were to in- 
vestigate the usability of the basic pathfinder/ 
notebook/log model currently implemented in 
Cyberdog. Additionally, featui^s which had not 
previously been tested, such as universal search, 
reading newsgroups, sorting items in the log, etc., 
were also investigated. 

The "Internet Toolkit" aspects of the product— 
Cyberbuttons, embedded Cyberitems, personal- 
ized pathfinders, etc. — were not testing during 
these sessions. Subjects who participated in these 
sessions will be brought into the lab in two weeks 
to participate in an investigation of those capabili- 
ties. 

Procedure 

Four subjects were tested during these sessions. All 
four were employees of Apple Computer. Three 
were moderately sophisticated users of the 
Macintosh and one was an experienced but unso- 
phisticated user. At least two had seen demonstra- 
tions of the Cyberdog software at some time in the 
past, although they did not appear that familiar 
with details of the product. 



The tester read a series of instructions to subjects in 
the Piaget's Kids usability lab, and then moved into 
the control room for the remainder of the test ses- 
sion. Subjects were given a script with a series of 
tasks and asked to read the instructions aloud, 
complete the tasks, and "think aloud" while carry- 
ing them out. They were able to communicate with 
the tester via microphone during the sessions. The 
subject's interaction with the software was record- 
ed on videotape for further review. Subjects typi- 
cally took about 30-45 minutes to complete the 
tasks. Following completion of the task the tester 
conducted a short debriefing. 

Prior to the test the Cyberdog software was set up 
in a folder called "Internet Access" which was lo- 
cated on the desktop. Within this folder, four items 
were displayed: "Internet Pathfinder", "Personal 
Notebook", "Other Pathfinders" (containing the 
Cyberdog "demo" pathfinders), and "Parts to 
Create PFinders" (including only cyberbutton sta- 
tionary and a clock part). This window is displayed 
on the next page. The latter two items were not 
used in the current testing. 

The script for the current testing is included in 
Appendix A. Subjects were required to explore the 
network, search for information on particular top- 
ics, move items to the notebooks and manipulate 
them there, work with the log, and find and read 
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postings in a particular discussion area. 



Results 



Overview 



In general, these testing sessions revealed that the 
usability level of the current Cyberdog software is 
very high. All four users were able to complete all 
of the tasks with little or no assistance from the 
tester. 



People had few problems with many aspects of the 
product: 

• All users went immediately to the Pathfinder 
during the exploration period, although one or 
two did mention that there was not indication 
about what they were supposed to click. 

• People had few problems navigating around 
the network, even learning to click on the "di- 
amonds" in the Web browser to move to 

other locations. 



The general pathfinder/ 
notebook/log model 
seemed to be understood 
by the people tested, al- 
though unsophisticated 
users may have some initial 
confusion about personal 
notebooks. 



38 items 



112.8 MB in disk 



5.8 MB available 



Internet Pathfinder 



Personal Notebook 



Other Pathfinders 



Also encouraging was the 
fact that all of the users ap- 
peared to enjoy using this 
software, and each was 
willing to return in a couple of weeks to perform 
more tasks. One user, who typically does not enjoy 
participating in usability testing, said at the end of 
the test "That wasn't so bad! 7 ' 

However, as described in the "Problems" section 
below there are many details which created some 
difficulty for most of those tested and which clearly 
need further design attention. Also, there are 
many other navigation and information location 
problems which were obvious during the tests but 
are more related to the way information is orga- 
nized on the Internet than to the Cyberdog soft- 
ware specifically. 



Good 



Parts to Create PFinders 
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• The three most 
sophisticated subjects 
had little problem un- 
derstanding the intent 
of the notebooks, al- 
though the least so- 
phisticated one tend- 
ed to move things to 
the desktop to save 
them for future refer- 
ence. 



People were, in 
general, able to manipulate items in the note- 
book pretty successfully, including creating 
and deleting categories, moving items from 
one category to another, and folding and un- 
folding category displays. There were prob- 
lems with naming categories, however, which 
are discussed in greater detail below. 

• People went naturally to the "Search" feature 
without directions to do so, and made success- 
ful use of it to find items they were asked to 
locate. 

• People understood and properly navigated the 
newsgroups and news postings windows. 



Problems 
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However, people also had some specific problems 
with various components of the software. The 
Universal Search feature created a number of such 
problems: 

• The "syntax" of the search string was decided- 
ly unclear for users. Small changes in specifica- 
tion (e.g. "Tasmania" versus "Tasmanian") re- 
turned quite different results. Also, users re- 
peatedly wanted to know how to limit the 
search and weren't clear to how to accomplish 
this. 

• The "or" nature of multiple words provided 
search results which were very confusing. For 
example, a search string of "Tasmanian Parks" 
returned, among other entries, information 
about "Tasmanian Sodomy Laws", which was 
remarked upon by one or two users. 

• The "batch return" of flocks of entries was dis- 
orienting to some users. That is, large groups 
of entries were returned in waves which visi- 
bly displace the entries people are looking at. 

• Users wanted feedback about how long a 
search was going to take, or more information 
about what was going on. One user wondered 
if a search across the whole Internet was going 
to take a very long time. 

• There was no obvious way to stop searches 
(which actually could be accomplished by clos- 
ing the search window). One subject suggested 
that "command-." should accomplish that task. 

Subjects also experienced some problems in inter- 
acting with the personal notebook. 

• When a new category was created, it appeared 
below the visible portion of the window, thus 
presenting no feedback that the category had 
been successfully located and confusing all 
users about where it was. 



• Subjects had great difficulty renaming cate- 
gories. Although most figured it out eventual- 
ly, it was not intuitive to double click on the 
name to put it in "edit mode." Typing a return 
when naming a category also caused the text 
previously typed in to disappear. At least two 
of the users appeared to always type a return 
after finishing typing the name they had se- 
lected. 

• Moving items from one category to the other 
was extremely difficult for users when both 
categories could not be made to appear in the 
window at the same time. This problem is due 
to the fact that the notebook windows do not 
presently autoscroll. 

• One user was confused between the "Set 
Default Notebook" and "Show Default 
Notebook" commands, repeatedly selecting 
the former when she intended the latter. 

• One user expressed the desire to have a confir- 
mation dialog when items, especially a catego- 
ry, are deleted (as, for instance, by dragging 
items to the trash). 

• The idea of "subscribing" to a newsgroup by 
putting it into the notebook was clear to some 
users but not all. 

• The idea of the notebook as a location for your 
"favorite places" may turn out to be a prob- 
lem for unsophisticated users. 

• One user was confused as to why there were 
lots of things in the default personal notebook 
before she put anything there, assuming the 
previous person had failed to "clean up." 

Although the log was used successfully by every- 
one, there were some significant problems: 
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• No one guessed what the "hand" in the log 
was supposed to represent. Comments includ- 
ed "What's this little hand?" and "I have no 
idea what the little hand in it means/' 

• People were not able to clearly articulate what 
the "hierarchical" display in the log was sup- 
posed to represent. 

Other components of the system provided some 
problems as well: 

• Users consistently double-clicked on items 
which require only a single click, such as 
Cyberbuttons and links in Web pages. 

• Users had consistent problems locating items 
in lists even when they were clearly visible in 
the window. 

• At least one user expressed discontent that 
window names often didn't match the name of 
the items which were opened to display them. 

• Users were consistently unhappy about the 
number of windows which, when opened, dis- 
played on partial lines of text rather than being 
opened to the full right edge. 

• The display of newsgroup labels was too far to 
the right in the window to be of much use to 
users searching for the "Bay Area" newsgroup 
collection. 

• The display of an folded "apple.com" news 
server in a window after the "Discuss" button 
was clicked gave every subject some initial 
hesitation. Subjects did not realize they were 
looking at a "news server." 

• The sudden shift from "page-oriented" Web 
browsers to "Finder-oriented" gopher 
browser was a little disorienting for at least 
one user (few actually ever got to a Gopher 



browser, spending most of their time on the 
World-Wide Web). 

• The most unsophisticated user was put off by 
URL specifications which she ran into in one 
Web page. When asked what she thought it 
was she said that it "looks like an IBM-type 
command." 

• Users tended to overuse the "New" item in 
the File menu when trying to create other ob- 
jects (such as categories), inadvertently creat- 
ing new notebooks and pathfinder documents. 
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Appendix A. Test Script 

Thanks for coming around today to help us. We're presently developing software which would allow users to find and re- 
trieve information on the world-wide Internet which you may have been hearing about. We have an early version of this 
software and we're interested in seeing how you might go about using it. 

You'll be asked to perform a number of tasks; please attempt to perform them in the best way you can. Don't worry about 
whether you can complete the tasks or not; just do your best. We're not testing you; we're just trying to determine how we can 
improve the software to make it as easy to use as possible. 

Also, as you go about performing these tasks, would you please speak your thoughts out loud? We'll be recording your actions 
and your voice and this will help us understand why you're doing what you're doing. 

Finally, let me mention again that what you're working with is software which is still early in the development phase, not 
the final product. Many things will not work as they will in the final product; on occasion, the system may crash, but don't 
worry about that. I'll try to mention when something won't work correctly so you won't be misled or confused. 

There is one problem I will mention right up front. In most of the windows there is a bug in the underlying software which 
makes it necessary to click in the grow box in the lower right-hand corner of the screen before the information in the window 
can be selected or manipulated. I'll point this out if you forget about it. I'll be observing you in the next room, behind the 
glass; some other observers may also join me in there from time to time. I'll be able to hear you as well as speak to you, so 
don't hesitate to ask if you have any questions. 

Do you have any questions right now before we get started? 
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Please read these instructions aloud and perform the tasks 
detailed below. When you get to the small hand symbol 
( D ® > ) this means you should go ahead and perform the task 
you've been asked to do, so please don't read beyond that 
place until you've attempted to complete that instruction. 

Task 1 : Look Around 

The materials you'll need to interact with the Internet are 
contained in the folder called "Internet Access." That win- 
dow is displayed on the screen in front of you. 

Before asking you do to any specific tasks, we'd like you to 
just look around the Internet using these tools. Go ahead and 
do that for about 10 minutes. Please note again that this is 
only a prototype and so it won't be possible to do many of the 
things you'd ordinarily like to do. (®*) 



this time. ( c ^) 

Task 3: Exploring the Internet 

Sometimes you may go out onto the Internet to look for some- 
thing specific, as you have been doing with the last task. 
Other times, you may simply want to wander around to see 
what's available. ( c ^ > ) 

Without using the "Search" button, look around the network 
a little bit to see if you can find some information about as- 
tronomy. If you find some, see if there's something about the 
recent collision of the comet Shoemaker-Levy with the 
planet Jupiter. ( B ® > ) 

Now see if you can find some information about travel, and 
take a look at it. (B®*) 



Task 2: Finding Information 

Please close all of the windows and go back to the Finder. 
Open the document you see in the Internet Access window 
called "Internet Pathfinder." You'll use this as your jumping- 
off point for your interaction with the Internet for the rest of 
the tasks. 

First, let's suppose you have an interest in parks of various 
kinds in foreign countries. In particular, you're interested in 
parks in Tasmania. Using the Internet Pathfinder, see if you 
can find if there is information on the Internet about 
Tasmanian parks. (^ > ) 

You are also quite interested in art, and in particular about 
painting. See if there is any information on the Louvre in 
Paris on the Internet. (^) 

Since you like art, and the Louvre in particular, it has oc- 
curred to you that you. might like to visit a couple of the 
places you have found on the Internet again and again to see 
what's new. See if you can figure out a convenient way of 
making it easy to go directly to one of these places again in 
the future without all of the navigation that was necessary 



Task 4: Tracking Where You've Been 

You have been moving around different places on the 
Internet during your session. Please find a way to display a 
list of all of the various places you have been. (*®* ) 

Please display this list in the order in which you have vis- 
ited the locations. (*® > ) 

Please display this list in alphabetical order. {*®) 

Please display this list in hierarchical order. Explain what 
this means (explain out loud). (^) 

Please return as quickly as possible to one of the places on 
the list. ("ST) 

Go back to the list again. Pick one of the items on the list, 
and do what's necessary to keep your own personal copy of 
the item so you can access the place it represents again di- 
rectly. 

Task 5: On-Line Discussions 
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# 



There is a place on the Internet where people engage in on- 
line conversations about a number of topics. Let's suppose you You're done! Thank you very much, 
are interested in the topic of bicycles in the San Franciso 
Bay area and would like to follow the items people are 
posting about that topic. Please find this on-line conference 
and read a couple of the postings there. ( 1 ® s ) 

Although you can't presently do this in this version of the 
software, please describe out loud how you think you might 
go about posting a reply or follow up message to the one you 
just read. 

Let's suppose that you think you're going to want to follow 
the discussions in this area. Do what you think is necessary 
* to "subscribe" or make it easy to access this topic in the fu- 
ture, so you don't have to navigate through other topics to 
find the Bay Area bicycles topic area. ) 

Task 6: Working with Your Favorite Places 

In some of the tasks you've already performed you've used a 
personal notebook to store items you wanted easy access to. 
Please display your default notebook now. (*®* ) 

All of the items in the category "Fun Stuff" are visible right 
now. Can you make it so that only the category title itself is 
visible in the window? (!®* ) 

There is an item in the notebook called "Apple Higher Ed 
Gopher Server." Can you find that item for me? (D®*) 

Please move the item "Apple Higher Ed Gopher Server" to 
the category "Starting Points." (®) 

There is an item called "ba.food" in the notebook. Please re- 
move it. (t®*) 

Please create a new category called "Pictures." Then move 
the items "Gallery" and "Image and Multimedia Library - 
Victoria Freeman" into it. ft®) 

Please delete the category "Work" along with all of its con- 
tents. (DS*) 
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